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Abstract Body 


Background / Context: 

We are presently in the fourth year of a six-year, lottery-based randomized control trial 
(RCT) studying the effectiveness of nine Colorado charter schools that have implemented Core 
Knowledge®. Core Knowledge (CK) is a comprehensive K-8 curriculum for language arts, 
math, science, social studies, visual arts, and music that is intended to build knowledge, 
concepts, and vocabulary systematically from one grade to the next, leading to substantial 
progress in reading (Hirsch, 2006). Rigorous experimental estimates of the impact of CK charter 
schools are of interest to education researchers and policymakers because these schools combine 
two growing influences on the landscape of U.S. education, each of which promises to improve 
student achievement, charter school organization and use of the CK curriculum. There are more 
than 50 CK charter schools in Colorado alone and well over 200 in the U.S. Yet there are no 
experimental studies of the effectiveness of these schools — studies that could shed light on the 
question of why particular kinds of charter schools work, if and when they do (Berends, Watral, 
Teasley, & Nicotera, 2008). 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this paper is to describe (1) a particular set of problems that arose in 
conducting the lottery-based RCT and (2) the solutions that we developed and implemented. We 
extract lessons of interest primarily to researchers conducting lottery-based studies. 

Setting / Population / Participants / Subjects: 

The nine participating CK charters are located in six school districts that range from the 
suburbs south of Denver to the city of Fort Collins in the north. The participants include two 
cohorts of students and their parents who participated in kindergarten entrance lotteries in 2010 
and 2011. Students are being tracked as they progress from kindergarten to third or fourth grade 
when they will take Colorado achievement tests in reading, mathematics, and writing in 2014 
and 2015. Cohort 1 (n = 887) is currently in second grade and Cohort 2 (n = 121 1) is in first 
grade. The treatment group in each cohort is defined as students who received an offer of 
admission, and the control group is defined as students who did not receive an offer of 
admission. Control group students attend an alternate school, which is most often a regular 
public (non-charter) school but may also be another charter or a private school. 

Significance / Novelty of study: 

Most lottery-based studies of the impact of charter schools on student achievement rely 
exclusively on test data that are routinely collected by states for accountability purposes (e.g., 
Gleason, Clark, & Tuttle, & Dwoyer, 2010; Abdulkadiroglu, Angrist, Dynarski, & Kane, 2011; 
Angrist, Dynarski, Kane, Pathak, & Walters, 2012). This approach has several important 
limitations. First, because states generally do not administer achievement tests until third grade, 
researchers interested in studying the impact of charter schools in the early grades must employ a 
posttest-only design and either recruit a larger sample of oversubscribed charter schools with 
lotteries or settle for less precision in their impact estimates. Second, the state tests are group- 
administered, which decreases reliability, and they do not assess many important forms of 
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academic knowledge (e.g., science) or skills that contribute to reading achievement (e.g., oral 
language and vocabulary). Third, parents who apply for admission to a charter school may 
enroll their child in a private school if they do not get an offer of admission. Because state test 
data are not available for students in private schools, there is differential attrition from the 
randomization sample with potential for bias. 

It is possible to overcome the above limitations with researcher-administered individual 
achievement pretests and/or posttests, but there are significant practical obstacles including: 1) 
obtaining district and school cooperation, especially for testing in alternate schools, 2) obtaining 
parent consent for student testing, especially pretesting prior to or immediately following the 
lottery before entering kindergarten, 3) ensuring comparable participation rates for students who 
do or do not receive an offer of admission, and 4) wide dispersion of the control group in 
alternate schools. We experienced all of these problems and eventually developed a "parent- 
centered" approach to solving them, as explained in the case study below. 

Proposed Method: The Prohlem(s) 

We were interested in the effectiveness of a particular kind of charter school in the early 
grades, and we were studying a relatively small number of elementary schools with small student 
populations. Most importantly, the CK curriculum emphasized science and social studies, but 
the state of Colorado did not assess science achievement until fifth grade and did not assess 
social studies at any grade level. For these reasons, our lES-funded proposal called for annual 
administration of achievement tests, beginning with a pretest prior to the lotteries and continuing 
through third grade, in addition to matching students’ lottery outcomes with state test scores. 

An initial difficulty was that we could not embed our parent/student recruitment and consent 
process for individual testing into the schools’ lottery application process as planned. The 
principals were unwilling to do this because they thought it would discourage applications to 
their school. Also, many of the lottery applications were received a short time before the lottery. 
As a result, our recruitment efforts involved mail and email delivery of materials and follow-up 
phone calls beginning two weeks prior to each lottery and continuing after the lottery for four to 
six weeks. For Cohort 1, the overall consent rate was unacceptably low (32%), and the 
treatment-control differential in response rate was atrocious (52% versus 25%). For Cohort 2, 
we obtained 220 preliminary interest forms before the lottery, solving the problem of differential 
consent through randomization, but that subgroup represented just 18% of the Cohort 2 lottery 
applicants. Thus for both cohorts we were unable to obtain a usable sample for pretesting. 

In retrospect, we should have realized that, for busy parents who are in the midst of getting 
their child enrolled in a charter or non-charter school for the coming school year, filling out a 
consent form so they can contribute to a 5-year research study is not a high priority. The timing 
of the distribution of our IRB -approved (and somewhat daunting, detailed and legalistic) consent 
form could not have been worse, especially for parents who had not received an offer of 
admission to one of the CK charters or did not expect to receive an offer (since several weeks 
may have passed since the lottery occurred). These parents may have perceived the study as 
advocacy for the very school that had not been forthcoming with an offer. They thought we were 
trying to prove that CK charters are effective. 

Although we would not have a pretest, our hopes were buoyed by the fact that, due to the 
addition of a second cohort, we would have a larger sample of students with state test scores 
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(over 2000 instead of the originally planned 1300) and might also have a larger sample of 
students for individually administered achievement tests. But we encountered a serious obstacle. 
When we submitted an application to conduct research in the county where approximately 60% 
of our control group students are enrolled in school, the application was rejected. This meant 
that we had to abandon our plan of testing control group students in the (mostly) regular public, 
non-charter schools where they were enrolled. 

Again in hindsight, it is perhaps not surprising that we failed to gain the cooperation of this 
district. Charter schools are controversial and unpopular with principals and teachers in non- 
charters. The school system administrators had no objection to our testing students in charter 
schools, but felt that testing in non-charters would be perceived as "trying to make them look 
bad." Our assurances that the reported results would be aggregated across multiple schools and 
districts had no effect. 

Based on these problems, we elected to forego the proposed individual testing for Cohort 1 
students in their kindergarten year (an intermediate posttest). However, we felt that if we chose 
to eliminate the individual testing altogether in subsequent years, there would be a substantively 
important negative consequence: Although we could still study the impact of assignment to a 
CK charter school on reading and mathematics as measured by the state tests given in grades 
three and four, we would lose the ability to answer several of our original research questions that 
asked about the impact of assignment to a CK charter school on students’ (1) word reading and 
decoding skills, (2) oral vocabulary and listening comprehension, and (3) knowledge of science 
and social studies content. All of these are emphasized by the CK curriculum but not tested by 
the state. Also, we had hypothesized that if there is a CK charter school effect on reading 
comprehension, it is mediated by growth in these skills, and we had included a research question 
to that effect. Our ability to test the meditational hypothesis would have been lost as well if we 
were forced to rely exclusively on the state tests. 

New Method: The Solution 

In the third year of the project we conducted a small parent survey to assess the feasibility of 
individually testing Cohort 1 students in a non-school setting. We offered a $50 gift card for 
participating in the survey. Results of this pilot survey indicated that 96% of the parents would 
be willing to transport their child to a non-school site for individual testing if they received a 
$150 gift card incentive. 

Encouraged by this result, we conducted a telephone survey of all Cohort 1 parents. A total 
of 552 parents responded, a 60% response rate, with a small treatment-control differential (62 vs. 
59%). What accounts for the dramatic improvement in both the overall response rate and the 
differential response? Several factors were likely to have influenced these results. First, the 
timing was better. When we began the parent survey, children were comfortably enrolled in first 
grade in a charter or regular public school, and the parents of children enrolled in alternate 
schools may have gotten over their disappointment. Second, we focused on parents (not 
students) and their experiences with the lottery application process. In conducting the parent 
survey, project staff introduced themselves and explained that they were calling about “a study 
that’s being done by researchers from the University of Virginia in cooperation with the 
Colorado Department of Education” (no mention of CK charter schools) that involved “parents 
like you, parents who applied to a charter school prior to kindergarten, so that we may better 
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understand your experience with the application process. We think it’s very important to 
understand the views of parents like you and we want to ask you some questions. . 

The survey items included reasons why parents apply to charter schools, the sources of 
information they used in applying, how they felt about the experience of applying, and their 
satisfaction with their child’s current school, charter or non-charter. Additionally, we asked 
parents to list all the schools they applied to, how they ranked them, how they responded to 
receiving or not receiving an offer of admission, and the reasons they made these choices (e.g., 
declining an offer because the school was too far from their home). The last few items, drawn 
from the pilot survey, explored the parent's willingness to transport the child to a non-school site 
for individual testing. 

Next, we obtained a waiver of university rules requiring the collection of social security 
numbers for gift cards over $50, allowing us to offer a $150 gift card as compensation. 

Finally, to schedule individual tests, we called all of the parents who had indicated on the 
survey that they would be willing to transport their child for testing. A high percentage (85%) of 
these parents did in fact bring their child to one of five non-school sites (e.g., a community 
recreation center or library) where we had rented facilities suitable for testing, and they received 
a $150 gift card. A total of 405 Cohort 1 students took achievement tests including Woodcock- 
Johnson (W-J) Letter and Word Identification, W-J Passage Comprehension, Peabody Picture 
Vocabulary Test (PPVT), and W-J Academic Knowledge, which includes science, social studies, 
and humanities. The participation rate was actually higher for control students than treatment 
students (47% of the full sample versus 44%). 

It is important to note two other facts. First, about 8% of the individually tested students are 
attending a private school. Second, the control group students are enrolled in more than 150 
different alternate schools, which would have made testing in the regular public schools a 
logistical nightmare. 

Conclusions: 

The parent survey not only gave us access to Cohort 1 students for testing but also proved to 
be a valuable source of data in its own right. In fact, although it was not included in our original 
proposal, we have come to view it as an integral part of the lottery-based RCT, indeed the key to 
our plans for Years 4-6. We will repeat the survey and testing process with Cohort 2. 

We think there are some lessons here for other charter school researchers who are 
contemplating the collection of achievement test data instead of relying on administrative data: 

1) Expect that schools will be unwilling or unable to embed parent and student recruitment 
in the lottery application process. Know that pre-lottery pretesting is probably infeasible. 

2) Expect a treatment-control differential if recruiting parents and students for pretesting 
after the lottery and prior to school entry. 

3) Be patient; wait a year or two, then talk with the parents. They are apt to respond to a 
survey that focuses on their experience in the charter application process, and they may 
also be willing to have their child tested at that point, regardless of whether or not they 
received an offer of admission. 

4) Expect that school districts will not support testing in alternate schools. Be prepared to 
implement costly (though logistically simpler) testing in non-school settings. 
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